Combined eye and gesture tracking

ABSTRACT

Method for controlling a combined eye and gesture tracking system ( 100 ), wherein the method comprises providing a graphical user interface ( 102 ) to a user including a plurality of graphical items ( 110 ) displayed on the graphical user interface ( 102, 300 ), displaying a graphical pointer ( 120 ) on the graphical user interface ( 102, 310 ), detecting a user eye gaze ( 210 ) associated with the eyes of the user ( 320 ), detecting a displacement user gesture ( 230 ) associated with a body part of the user ( 330 ), and controlling a rate of movement of the graphical pointer ( 120 ) on the graphical user interface based on the displacement user gesture ( 230 ) and the user eye gaze ( 210, 340 ).

TECHNICAL FIELD

Various embodiments relate to a method for controlling a combined eyeand gesture tracking system, and in particular to a method forcontrolling a combined eye and gesture tracking system for a graphicaluser interface in an automotive environment.

BACKGROUND

Use of electronic devices, such as navigation systems, communicationdevices, and media players, is becoming increasingly popular inautomotive environments. Such devices provide users with convenientaccess to informational content, such as navigational instructions,media content, such as music, news, communication services and means tocontrol the operation of a vehicle.

Human machine interfaces (HMI) are known to include graphical userinterfaces displaying a plurality of menu items. Menu items may relateto entries of a graphical menu of the graphical user interface which canbe selected or activated by the user using e.g. a graphical pointer.Upon said selection, a certain functionality may be accessed or invoked,e.g. a sub-menu and/or a function or application and/or a change ofsettings, etc. For example, menu items may relate to one or more of thefollowing: applications, audio and/or video tracks, system settings,pictures, downloaded items.

Often a plurality of such menu items is displayed on a display. A usermay select and activate a particular one of the plurality of menu items,depending on his selection choice, by controlling a graphical pointer onthe graphical user interface. For this, at one given point in time theplurality of menu items may be displayed on the display at leastpartially or entirely and the user may select one of the menu items fromthe plurality of menu items by moving the graphical pointer to one ofthe menu items using e.g. mechanical buttons, a touch controller, touchdisplay, independent eye tracking or independent gesture trackingsystems.

In situations as set forth above, in particular independent eye trackingtechnology and independent gesture tracking technology offer to the userthe potential for convenient interacting with a graphical userinterface. However, such techniques face certain limitations: thecontrol options provided by an eye or gesture tracking system may not besatisfactory. Furthermore speed and precision of the eye or gesturetracking system may be limited.

SUMMARY

Accordingly there is a need to provide a method and system for combinedeye and gesture tracking for a graphical user interface, which providesimproved control to a user.

According to an embodiment, a method for controlling a combined eye andgesture tracking system is provided, wherein the method comprises thefollowing steps: Providing a graphical user interface to a userincluding a plurality of graphical items displayed on the graphical userinterface, displaying a graphical pointer on the graphical userinterface, detecting a user eye gaze associated with the eyes of theuser, detecting a displacement user gesture associated with a body partof the user, and controlling a rate of movement of the graphical pointeron the graphical user interface based on the displacement user gestureand the user eye gaze.

Thereby, a method for controlling a combined eye and gesture trackingsystem is provided, that may have increased precision and increasedspeed in the control by a user. The response speed of the combined eyeand gesture tracking system may be increased, e.g., if compared toconventional eye or gesture tracking systems. Thus, a faster and moreprecise access to graphical user interface elements may be provided tothe user, which, in an automotive environment, typically increases thesafety of operation of a vehicle. Further, the design and production ofa vehicle can be realized easier and with less cost with a greaterfreedom in placing a screen within the vehicle compared to conventionaleye or gesture tracking systems. By controlling a rate of movement ofthe graphical pointer on the graphical user interface based on thedisplacement user gesture and the user eye gaze, a possible misalignmentcalibration error of the eye tracking may be corrected by continuouslytracking of the displacement user gesture direction and accordinglyadjusting of the displacement direction of the graphical pointer withoutthe user's notice. In particular, in such a case of a misalignmentcalibration error of the eye tracking system, the user may look at aparticular graphical item, but due to the misalignment calibration errorthe eye tracking system may recognize his eye gaze not as correlatedwith the particular graphical item, but with a position besides theparticular graphical item, then the user would have to look to aposition besides the particular graphical item to the other direction,in order to correct the misalignment calibration error. By continuouslytracking of the displacement user gesture direction and accordinglyadjusting of the displacement direction of the graphical pointer withoutthe user's notice, the user does not have to look besides the graphicalitem to select the graphical item. Additionally, moving a graphicalpointer in a translational movement on a screen may be more intuitive bya displacement user gesture than by a user eye gaze.

Further, the method can comprise detecting that the displacement usergesture corresponds to one of a plurality of predetermined displacementuser gestures associated with a body part of the user, wherein the bodypart of the user is moved in a displacement direction relative to thegraphical user interface, and controlling a rate of movement of thegraphical pointer can comprise moving the graphical pointer with avariable rate of movement in the displacement direction to a positionassociated with one of the plurality of graphical items taking intoaccount the user eye gaze. That the displacement user gesturecorresponds to one of a plurality of predetermined displacement usergestures may provide the effect that the graphical pointer can't beaccidentally displaced, if the user makes an arbitrary moving usergesture. This may increase the precision and safety of the combined eyeand gesture tracking system. Such controlling a rate of movement of thegraphical pointer may additionally increase the speed, precision andaccess rate of the combined eye and gesture tracking system.

Further, the graphical pointer can be selectively moved based on acorrelation of the displacement user gesture with the user eye gaze,thereby further increasing the precision and speed of the combined eyeand gesture tracking system.

Further, detecting a user eye gaze can comprise determining that theuser is looking at the graphical user interface, and determining an eyefocus position on the graphical user interface, wherein the graphicalpointer is moved only if the displacement direction is directed in adirection where the eye focus position is located. That the graphicalpointer is moved only if the displacement direction is directed in adirection where the eye focus position is located further increases theprecision, speed and safety of the combined eye and gesture trackingsystem.

Further, controlling a rate of movement of the graphical pointer cancomprise comparing the displacement direction with the eye focusposition, and moving the graphical pointer with a higher speed, only ifthe displacement direction correlates with the eye focus position.Moving the graphical pointer with a higher speed, only if thedisplacement direction correlates with the eye focus position furtherincreases the precision, speed and safety of the combined eye andgesture tracking system.

Further, each graphical item can comprise a predetermined snap-in areaaround the graphical item, wherein a rate of movement of the body partleads to a smaller rate of movement of the graphical pointer when thegraphical pointer is within the predetermined snap-in area and thedisplacement direction is not substantially in direction to thegraphical item, compared to a rate of movement of the graphical pointerwhen the graphical pointer is within the predetermined snap-in area andthe displacement direction is substantially in direction to thegraphical item or when the graphical pointer is not within thepredetermined snap-in area. The snap-in area provides a furtherincreased precision and speed and more intuitive usability of thecombined eye and gesture tracking system, wherein the graphical pointertends to be held in position over a menu item, respectively is moreeasily stopped, once it has reached the graphical target item.

Further, the method for controlling a combined eye and gesture trackingsystem can comprise detecting an initiation user gesture, determining ifthe initiation user gesture is substantially equal to a predeterminedinitiation user gesture, wherein the steps of displaying a graphicalpointer and controlling a rate of movement of the graphical pointer areperformed only if the initiation user gesture is substantially equal tothe predetermined initiation user gesture. That the steps of displayinga graphical pointer and controlling a rate of movement of the graphicalpointer are performed only if the initiation user gesture issubstantially equal to the predetermined initiation user gestureprovides the effect of preventing that the user accidentally activatesfunctions or moves the graphical pointer implicitly. Thereby, gestureprocessing logic is improved by providing faster and more precise accessto the graphical elements. Therefore, the user can look at all itemswithout visible action of the Human Machine Interface (HMI), e.g. movingthe graphical pointer, highlighting certain graphical items, activatingcertain graphical items. In particular, if eye focus position isdisplayed to the user, a risk of jumping of the displayed eye focusposition on the graphical user interface is avoided.

Further, displaying the graphical pointer can comprise determining aninitiation position of the graphical pointer on the graphical userinterface, and displaying the graphical pointer at the initiationposition. By displaying the pointer at the determined initiationposition, the reaction time of the combined eye and gesture trackingsystem is reduced, as the graphical pointer is easier controlled fromthe initiation position by the user than from an arbitrary position onthe graphical user interface.

Further, the initiation position of the graphical pointer can be the eyefocus position on the graphical user interface, or the position on thegraphical user interface, which the user has been looking at for thelongest time within a predetermined time range, or the position on thegraphical user interface, which the user has been looking at with thehighest count of refocusing actions within a predetermined time range,or a median of determined eye focus positions that have been determinedwithin a predetermined time range. Thereby, an initial position of thegraphical pointer is chosen, that corresponds to the user's actions andneeds at the moment of displaying the graphical pointer, thus thereaction time of the combined eye and gesture tracking system isreduced, as the graphical pointer is easier controlled from theinitiation position by the user than from an arbitrary position on thegraphical user interface. Furthermore, the initiation position can be apredetermined position on the graphical user interface, a so-called homeposition, which can be typical for each menu page and therefore iseasily recognized by the user.

Further, the method for controlling a combined eye and gesture trackingsystem can comprise detecting an engagement user gesture associated withthe body part of the user, determining, if the engagement user gestureis substantially equal to a predetermined engagement user gesture, andselecting the one of the plurality of graphical items associated withthe position of the graphical pointer, if the engagement user gesture issubstantially equal to a predetermined engagement user gesture.Selecting the one of the plurality of graphical items associated withthe position of the graphical pointer, if the engagement user gesture issubstantially equal to a predetermined engagement user gesture has theeffect of preventing that the user accidentally activates functions.Thereby, gesture processing logic is improved by providing faster andmore precise access to the graphical elements and safety of the vehicleoperation is increased.

Further, the method for controlling a combined eye and gesture trackingsystem can comprise setting the graphical pointer to a locked stateduring a transformation between user gestures of the body part or duringthe execution of a user gesture that is not authorized to change theposition of the graphical pointer in such a way, that during the lockedstate, the position of the graphical pointer cannot be substantiallychanged from the last valid position of the graphical pointer until auser gesture is recognized, which is authorized to change the positionof the graphical pointer. Setting the graphical pointer to a lockedstate has the effect of preventing that the user accidentally moves thegraphical pointer. Therefore, the position of the graphical pointerremains unchanged during transition of gestures, during change of HMIdisplay, providing the same user feeling as on a traditional desktopcomputer. Thereby, gesture processing logic is improved by providingfaster and more precise access to the graphical elements.

Further, the body part can comprise at least one of a finger, inparticular index finger, part of a hand, a hand, part of an arm, an arm,or part of a shoulder of the user, having the effect, that user has morefreedom to move while controlling the combined eye and gesture trackingsystem and can thus control the vehicle in a safer manner.

Further, the method for controlling a combined eye and gesture trackingsystem can comprise detecting, that a user gesture made by the user isnot substantially equal to one of the group of predetermined initiationgesture, predetermined displacement gesture or predetermined engagementgesture, and removing the graphical pointer from the graphical userinterface. By removing the graphical pointer from the graphical userinterface if it is detected that a user gesture made by the user is notsubstantially equal to one of the group of predetermined initiationgesture, predetermined displacement gesture or predetermined engagementgesture, the user does not have to terminate the process of control by aseparate user action and thus the vehicle can be controlled in a saferway.

According to a further embodiment, a combined eye and gesture trackingsystem is provided comprising at least one camera, at least onegraphical user interface, a memory configured to store program code, atleast one processor coupled with the memory and the at least onegraphical user interface and configured to execute the program code,wherein execution of the program code causes the at least one processorto perform the following: Displaying a plurality of graphical items onthe graphical user interface to a user, displaying a graphical pointeron the graphical user interface, detecting a displacement user gestureassociated with a body part of the user, detecting a user eye gazeassociated with the eyes of the user, and controlling a rate of movementof the graphical pointer on the graphical user interface based on thedisplacement user gesture and the user eye gaze.

Thereby, a combined eye and gesture tracking system is provided that hasincreased precision and speed for the control by a user, wherein theresponse speed of the combined eye and gesture tracking system may beincreased and a faster and more precise access to HMI elements may beprovided to the user compared to conventional eye or gesture trackingsystems. Further, the design and production of a vehicle can be realizedeasier and with less cost with a greater freedom in placing a screenwithin the vehicle compared to conventional eye or gesture trackingsystems. By controlling a rate of movement of the graphical pointer onthe graphical user interface based on the displacement user gesture andthe user eye gaze, a possible misalignment calibration error of the eyetracking may be corrected by continuously tracking of the displacementuser gesture direction and accordingly adjusting of the displacementdirection of the graphical pointer without the user's notice. Inparticular, in such a case of a misalignment calibration error of theeye tracking system, the user does not have to look besides thegraphical item to select the graphical item. Additionally, moving agraphical pointer in a translation movement on a screen is moreintuitive by a displacement user gesture than by a user eye gaze.

Although specific features described in the above summary and thefollowing detailed description are described in connection with specificembodiments and aspects of the present disclosure, it should beunderstood that the features of the exemplary embodiments and aspectsmay be combined with each other unless specifically noted otherwise.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure will now be described in more detail withreference to the accompanying drawings.

FIG. 1 illustrates a schematic drawing of a combined eye and gesturetracking system according to embodiments of the present disclosure;

FIG. 2 schematically illustrates a combined eye and gesture trackingsystem including a graphical user interface displaying a graphicalpointer with indicated eye focus position according to embodiments ofthe present disclosure;

FIG. 3 schematically illustrates the graphical user interface of FIG. 2with indicated eye focus position with an active gravity effect;

FIG. 4 schematically illustrates the graphical user interface of FIG. 3with indicated displacement direction with active gravity effect;

FIG. 5 schematically illustrates an initiation user gesture according toembodiments of the present disclosure;

FIG. 6 schematically illustrates a displacement user gesture accordingto embodiments of the present disclosure;

FIG. 7 schematically illustrates an engagement user gesture according toembodiments of the present disclosure;

FIG. 8 schematically illustrates the graphical user interface of FIG. 2with locked graphical pointer;

FIG. 9 illustrates a flowchart of a method for controlling a combinedeye and gesture tracking system according to embodiments of the presentdisclosure;

FIG. 10 illustrates a flowchart of a method for controlling a combinedeye and gesture tracking system according to an example;

FIG. 11 illustrates a flowchart of a method for controlling a combinedeye and gesture tracking system according to another example;

DETAILED DESCRIPTION OF THE EMBODIMENTS

In the following, concepts in accordance with exemplary embodiments willbe explained in more detail and with reference to the accompanyingdrawings.

The drawings are to be regarded as being schematic representations andelements illustrated in the drawings are not necessarily shown to scale.Rather, the various elements are represented such that their functionand general purpose become apparent to a person skilled in the art. Anyconnection or coupling between functional blocks, devices, components,or other physical or functional units shown in the drawings or describedherein may also be implemented by an indirect connection or coupling. Acoupling between components may also be established over a wirelessconnection. Functional blocks may be implemented in hardware, firmware,software, or a combination thereof.

Hereinafter, various techniques with respect to employing a combined eyeand gesture tracking system are described. In some examples, thecombined eye and gesture tracking system may be employed in a vehiclesuch as a passenger car.

FIG. 1 illustrates a schematic drawing of a combined eye and gesturetracking system 100 according to embodiments of the present disclosure.The combined eye and gesture tracking system 100 comprises a camera 101,a graphical user interface 102, a memory 103, and a processor 104. Thememory 103 may be non-volatile. The combined eye and gesture trackingsystem 100 further comprises a Human Machine Interface (HMI) comprisingthe camera 101 and the graphical user interface 102. Generally, theHuman Machine Interface (HMI) may comprise more than one camera 101 andmore than one graphical user interface 102. The Human Machine Interface(HMI) may comprise elements such as: a touch screen, voice control, akeypad, and/or steering wheel buttons, etc.

Different mounting positions of the camera 101 are conceivable. One ormore cameras 104 may be mounted below the instrument cluster, e.g.,horizontally centered below the instrument cluster with 0 degreehorizontal angle, shifted forward a towards the driver with a slightlyhigher vertical angularity (towards the face of an average size person)than the instrument cluster. The vertical angle of the camera could alsobe linked to the steering wheel adjustment. When presuming that a driveris responsible for setting up a reasonable steering wheel adjustmentregarding ergonomics and a clear view on the instrument cluster, thenpossibly a static mounting position (regarding depth and vertical angle)is sufficient for covering a wide range of sizes of drivers. One or morecameras 101 may be mounted on the instrument cluster hood. Horizontallycentered on the instrument cluster hood with 0 degree horizontal angle,possibly shifted backward towards the windshield with a suitablevertical angle (towards the face of an average size person). Thismounting position could enable the combined eye and gesture trackingsystem for use with head-up systems. One or more cameras 101 may bemounted near an A-pillar (driver side). One or more cameras 101 may becentered on the dashboard. One or more cameras 104 may be mounted belowthe central display. Generally, any other position giving adequateperformance/precision is available for mounting one or more cameras 101.

A suitable camera can be an infrared camera. Here, bright-pupil trackingcan be used for reliable eye-tracking systems. By usinginfrared/near-infrared light as illumination source, and even morereliable eye tracking can be implemented. Also passive infrared camerasmay be employed. The illumination source, if available, can be activewhenever eye or gesture tracking is enabled, even during daytime.Available eye or gesture tracking solutions based on camera, controlunit, and software may detect and adapt discretely to changes in thelighting conditions. As infrared/near-infrared light is invisible forthe human eye, it doesn't cause distraction.

The processor 104 of the combined eye and gesture tracking system 100 isconfigured to implement techniques of eye and gesture tracking. E.g.,the processor 104 may be configured to execute the method as illustratedby one of the flowcharts of FIGS. 9 to 11.

For effective operation and control by a user, Human machine interfaces(HMI) can include graphical user interfaces displaying a plurality ofmenu items. Menu items may relate to entries of a graphical menu of thegraphical user interface which can be selected or activated by the userusing a graphical pointer. A graphical pointer corresponds to a moveableitem, like a mouse pointer, a target cross, a spotlight or the like, allof which are representing the current HMI focus of the system, displayedon the graphical user interface 102. The graphical pointer correspondsto a position on the graphical user interface 102. A user may select andactivate a particular one of the plurality of menu items beneath thegraphical pointer depending on his selection choice, by using thegraphical pointer on the graphical user interface. For this, at onegiven point in time the plurality of menu items may be displayed on thegraphical user interface at least partially or entirely and the user mayselect one of the menu items from the plurality of menu items bycontrolling the graphical pointer to a position corresponding to one ofthe menu items using e.g. hardware buttons, a touch display, touchcontroller, eye tracking or gesture tracking.

A user eye gaze can correspond to: the user is looking at a given pointof the display, the eye-tracking system recognizes/calculates tocoordinates and optionally the level of variance/inaccuracy of thevisual focus and passes the data to a processor.

The graphical user interface may be depicted on one or more displays.E.g., the graphical user interface may be depicted on a clusterinstrument display of a vehicle, a center console display, aninfotainment display, and/or a head-up display (HUD). Also, threedimensional displays, e.g. autostereoscopic displays, are conceivable.

Where a given menu item is activated, a corresponding functionassociated with the respective menu item may be activated or triggered.Activation of the function may comprise one or more of the following:triggering display of additional information, e.g., in a reserved areaof the graphical user interface and/or nearby the activated menu itemusing, e.g., a pop-up window. E.g., a function can be any audio-visualreactions such as a pop-up menu being displayed, a menu being browsed,etc. Additional information may also be displayed within the menu item,e.g., by changing its shape or form. Alternatively or additionally, afunction can correspond to a non-visual reaction which changes theinternal state of a vehicle component. In some examples, the functioncan correspond to logging the last coordinate of the orientation of theeye gaze. Activation of the function associated with a given menu itemmay also comprise changing one more operational parameters of thevehicle or of a vehicle component, e.g., a volume of an entertainmentsystem, wiper speed, temperature of the air conditioning system, etc.

The techniques described herein may be applicable to driver and/orcodriver control of a vehicle-based eye-tracking system. One or morecameras may define one or more fields of view which include the driverseating area and/or the codriver seating area. If the driver andcodriver are enabled to operate the same physical display (e.g. of theinfotainment system—usually a central display), then it may be desirableto provision a split view display, which offers e.g. two independentHuman Machine Interfaces (HMI) for driver and codriver, e.g., separatedby viewing angle. Alternatively or additionally, it may be possible toimplement a “master/slave” logic to solve access conflicts betweendriver and codriver.

FIG. 2 illustrates a combined eye and gesture tracking system 100including a graphical user interface 102 displaying a graphical pointer120 at an initiation position of the graphical pointer 122 and anindicated eye focus position 130 according to embodiments of the presentdisclosure. The initiation position may also be referred to as seedposition. For example, the graphical pointer 120 may be located at theinitiation position 122 when re-setting the system, e.g., after atimeout of user inactivity, etc. For example, the graphical pointer 120may be located at the initiation position 122 in response to detectingan initiation user gesture. For example the initiation position of thegraphical pointer can be the eye focus position on the graphical userinterface, or the position on the graphical user interface, which theuser has been looking at for the longest time within a predeterminedtime range. Furthermore, the initiation position can be the position onthe graphical user interface, which the user has been looking at withthe highest count of refocusing actions within a predetermined timerange, or a median of determined eye focus positions that have beendetermined within a predetermined time range. Additionally, theinitiation position can be a predetermined position on the graphicaluser interface, a so-called home position. For example, the graphicalpointer 120 may be located at the initiation position 122 if the systeminitiates in a predefined state which does not depend on the eye gazeand/or the user gesture. Thereby, the position at which the graphicalpointer is displayed may be typical for each of a plurality of menupages and therefore may be easily recognized by the user.

Further, on the graphical user interface 102 a plurality of graphicalitems 110 are displayed. The indicated eye focus position 130 is notvisible to the user, in other cases it can be visible to the user.

FIG. 3 illustrates the graphical user interface 102 of the combined eyeand gesture tracking system 100 of FIG. 2 displaying a graphical pointer120 at an initiation position of the graphical pointer 122 and aplurality of graphical items 110 according to embodiments of the presentdisclosure. Therein, the indicated eye focus position is displayed aseye focus position with an active gravity effect 131. Here, the user hasfocused the graphical item (“NAME”) by an eye gaze, but has not yet madea substantial displacement user gesture, e.g. by moving the indexfinger. The eye focus position with an active gravity effect 131 is notvisible to the user, in other cases it can be visible to the user.Generally, the displacement user gesture may correspond to movement of abody part in free space. For example, the displacement user gesture mayrelate to moving an arm and/or finger of the user. In accordance withgesture recognition systems known in the art, a gesture can be detectedby a camera, or another contactless optical detection device. Therein, agesture is a motion of the body, or a body part, that containsinformation, e.g. waving goodbye. Whereas pressing a key on a keyboard,or moving a mouse is not a gesture because the motion of a finger on itsway to hitting a key, or the motion of the hand while moving the mouseis neither observed nor significant. All that matters is which key waspressed, or the movement of the mouse.

FIG. 4 illustrates the graphical user interface of the combined eye andgesture tracking system 100 of FIG. 3 displaying a graphical pointer 120at an initiation position 122 of the graphical pointer 120, a pluralityof graphical items 110, and an eye focus position with an active gravityeffect 131 according to embodiments of the present disclosure. Therein,a displacement direction 231 of the graphical pointer 120 is furtherdisplayed. The displacement direction 231 of the graphical pointer mayor may not be visible to the user.

FIG. 5 illustrates an initiation user gesture 220 according toembodiments of the present disclosure. Therein, the initiation usergesture is a gesture or position of a hand of a user, wherein allfingers are closed except the index finger, which is substantiallyextended to is full length. This is an example. In other examples, theinitiation user gesture 220 can be a gesture or position of any bodypart of the user, comprising one or more fingers, in particular an indexfinger, a part of a hand, a hand, a part of an arm, an arm, shoulder,part of a shoulder or head of the user. Detection of the initiation usergesture 220 may result in the graphical pointer 120 being depicted atthe initiation position 120.

FIG. 6 illustrates a displacement user gesture 230 according toembodiments of the present disclosure. Therein, the displacement usergesture 230 is a gesture or movement of a hand of a user, wherein allfingers are closed except the index finger, which is substantiallyextended to its full length, wherein the positions of the fingers toeach other remain substantially equal, only the hand as a whole makes asubstantially translational movement. The displacement user gesture 230can be a gesture or position of any body part of the user, comprisingone or more fingers, in particular an index finger, a part of a hand, ahand, a part of an arm, an arm, shoulder, part of a shoulder or head ofthe user. The displacement user gesture 230 can comprise a sampledanchor, e.g. the tip of the index finger or another tip of anotherextended finger, of the displacement user gesture 230. Further, themovement of the graphical pointer 120 can be controlled by a direct linkbetween the sampled anchor of the displacement user gesture 230 and themovement of the graphical pointer 120. Therein, the direction and rateof movement of the graphical pointer 120 are directly correlated withthe direction and rate of movement of the displacement user gesture 230,e.g. the direction a rate of movement of the sampled anchor. Theillustrated displacement user gesture 230 is advantageous, as theposition or movement of the tip of the index finger can easily andprecisely be tracked by the combined eye and gesture tracking system100.

FIG. 7 illustrates an engagement user gesture 260 according toembodiments of the present disclosure. Therein, the engagement usergesture 260 is a gesture or movement of a hand of a user, wherein allfingers are closed except the index finger, which is substantiallyextended to its full length, and wherein the index finger moves towardsthe balm of the hand. The engagement user gesture 260 can be a gestureor position of any body part of the user, comprising one or morefingers, in particular an index finger, a part of a hand, a hand, a partof an arm, an arm, shoulder, part of a shoulder or head of the user. Theengagement gesture 260 can comprise the sampled anchor of thedisplacement user gesture 230.

FIG. 8 illustrates the graphical user interface 102 of FIG. 2 withlocked graphical pointer 121. In a locked state of the graphical pointer121 during a transformation between user gestures of the body part orduring the execution of a user gesture that is not authorized to changethe position of the graphical pointer, the position of the graphicalpointer cannot be substantially changed from the last valid position ofthe graphical pointer until a user gesture is recognized whichauthorizes change of the position of the graphical pointer. The lockedstate of the graphical pointer 121 is not necessarily visually indicatedto the user. In the example of FIG. 8, the user is engaging thegraphical item 110 beneath the current eye focus position 130 by anengagement gesture 230—therefore the position of the graphical pointer121 is temporarily locked.

FIG. 9 illustrates a flowchart of a method for controlling a combinedeye and gesture tracking system 100 according to embodiments of thepresent disclosure. The method is started, then a graphical userinterface 102 is provided to a user, wherein a plurality of graphicalitems 110 on the graphical user interface 102 are displayed to the userin step 300. In step 310, a graphical pointer 120 is displayed on thegraphical user interface 102. Therein, displaying the graphical pointercan comprise animation such as fading in of the graphical pointer. Instep 320, a user eye gaze 210 associated with the eyes of the user isdetected. The step 320 of detecting a user eye gaze 210 can comprisecalculating a user eye gaze from user eye gaze data, which is obtainedby continuously tracking the eyes of a user. The step 320 of detecting auser eye gaze 210 may not be indicated to the user. In step 330, adisplacement user gesture 230 associated with a body part of the user isdetected. For example, recognition of a displacement user gesture 230may trigger a system action visible for the user such as movement of thegraphical pointer 120. For example, the displacement user gesture 230may trigger the movement of the graphical pointer 120. In step 340, arate of movement of the graphical pointer 120 on the display of thegraphical user interface 102 is controlled based on the displacementuser gesture 230 and the user eye gaze 210. Therein, the rate ofmovement of the graphical pointer can correlate directly to the rate ofmovement of the displacement user gesture 230. In other examples, therate of movement of the graphical pointer 120 can be faster or slowerthan the rate of movement of the displacement user gesture 230. Inanother example the rate of movement of the graphical pointer 120 can bea variable movement, in particular the rate of movement can increase ordecrease during the movement, or can be slower respectively faster thanthe rate of movement of the displacement user gesture 230 at thebeginning or end of the movement of the graphical pointer 120, based onthe user eye gaze 210. In step 340 of controlling a rate of movementbased on displacement user gesture 230 and the user eye gaze 210, theuser eye gaze 210 can be a correlated user eye gaze, wherein thecorrelated user eye gaze is temporally correlated to the displacementuser gesture. In particular, the correlated user eye gaze can bedetected at the same time as the displacement user gesture 230. Themethod ends after step 340.

FIG. 10 illustrates a flowchart of a method for controlling a combinedeye and gesture tracking system 100 according to another example. Themethod starts and in step 300 a graphical user interface 102 is providedto a user. In step 330, an initiation user gesture 220 is detected. Thestep 330 of detecting the initiation user gesture 220 can furthercomprise deciding if the initiation user gesture 220 is within in avalid spatial zone for recognition, wherein the initiation user gesture220 is only confirmed to be present if the initiation user gesture 220is within the valid special zone for recognition. Then it is decided ifthe initiation user gesture 220 is substantially equal to one of aplurality of predetermined initial user gestures. If it is decided thatinitiation user gesture 220 is not substantially equal to one of aplurality of predetermined initial user gestures, then the process ends.Therefore, only when the user gesture corresponds to one of a pluralityof initiation user gestures, the graphical pointer 120 is controlled. Ifit is decided that the initiation user gesture 220 is substantiallyequal to one of a plurality of predetermined initial user gestures, thenit is decided if the user is looking at the graphical user interface102. If it is decided that the user is not looking at the graphical userinterface 102, then in step 330 a graphical pointer 120 is displayed inthe middle, center or at a predetermined home position of the graphicaluser interface 102. Then, in step 340, a rate of movement of thegraphical pointer 120 is controlled. If it is decided that the user islooking at the graphical user interface 102, then an eye focus position130 is determined. After that, in step 310, a graphical pointer 120 isdisplayed on the graphical user interface 102 based on the eye focusposition 130. In step 340, a rate of movement of the graphical pointer120 is controlled.

The method can comprise detecting that the displacement user gesture 230corresponds to one of a plurality of predetermined displacement usergestures associated with a body part of the user, wherein the body partof the user is moved in a displacement direction 231 relative to thegraphical user interface 102. Controlling a rate of movement of thegraphical pointer 102 can comprise moving the graphical pointer 102 witha variable rate of movement in the displacement direction 231 to aposition associated with one of the plurality of graphical items takinginto account the user eye gaze. Further, controlling a rate of movementof the graphical pointer 120 can comprise comparing the displacementdirection 231 with the eye focus position 130, and moving the graphicalpointer with a higher speed, if or only if the displacement direction231 correlates with the eye focus position 130. Moving the graphicalpointer 120 with a higher speed, only if the displacement direction 231correlates with the eye focus position 130 further increases theprecision and speed of the combined eye and gesture tracking system.This so-called virtual gravity effect can be switched on or off based onthe correspondence of displacement user gesture 231 and eye focusposition 130. When continuous eye tracking data is available, acontinuously calculated eye focus position can be checked against thedisplacement user gesture 230 by a further algorithm continuously. Thegravity effect can accelerate when the graphical pointer 120 gets closertowards the eye focus position 130. The gravity effect may get strongeras the gap is closing down to a graphical target item and may even havea snap-in effect within a certain range of the graphical target item,wherein the movement is further accelerated towards the graphical targetitem and is stopped, once it has reached the graphical target item.

In case the system does not recognize a correlation, e.g. the gesturemoves in the opposite direction to the eye focus position 130, theacceleration effect is reduced or disabled at all—so the graphicalpointer 120 is solely controlled by the displacement user gesture—untilthe combined eye and gesture tracking system 100 recognizes a validcorrelation again between eye tracking data (eye user gaze) anddisplacement user gesture 230, which would enable or increase theacceleration effect again.

Further, each graphical item 110 can comprise a predetermined snap-inarea around the graphical item 110, wherein a rate of movement of thebody part leads to a smaller rate of movement of the graphical pointer120 when the graphical pointer 120 is within the predetermined snap-inarea and the displacement direction 231 is not substantially indirection to the graphical item 110, compared to a rate of movement ofthe graphical pointer 120 when the graphical pointer 120 is within thepredetermined snap-in area and the displacement direction 231 issubstantially in direction to the graphical item 110 or when thegraphical pointer is not within the predetermined snap-in area. Thesnap-in area further increases the precision and speed of the combinedeye and gesture tracking system, wherein the graphical pointer 120 tendsto be held in a position over a graphical item. The method can furthercomprise the step of controlling the graphical pointer 120 to one of theplurality of graphical items 110 and activating a corresponding functionassociated with the one of the plurality of graphical items 110. Theprocess is ended after that.

Additionally, the method for controlling a combined eye and gesturetracking system 100 can comprise detecting that a user gesture made bythe user is not substantially equal to one of the group of predeterminedinitiation gestures, predetermined displacement gestures orpredetermined engagement gestures, and/or detecting that the usergesture is not valid any more, e.g. the user has removed the body partfrom the valid spatial zone for recognition or the user has started anew user gesture which invalidates the previous user gesture and partsof its data, and accordingly removing the graphical pointer 120 from thegraphical user interface 102. Removing the graphical pointer 120 cancomprise fading the graphical pointer 120 out.

FIG. 11 illustrates a flowchart of a method for controlling a combinedeye and gesture tracking system according to another example. First, auser gesture is detected. Then, it is decided if the user gesture isrecognized. If the user gesture is not recognized, the process ends. Ifthe user gesture is recognized as a displacement user gesture 230, thenan eye focus position 130 is tracked. After that, in step 340, a rate ofmovement of the graphical pointer 120 is controlled based on the usergesture and the eye focus position. If the user gesture is recognized asan engagement user gesture 260, then a corresponding function isactivated.

LIST OF REFERENCE SIGNS

100 combined eye and gesture tracking system

101 camera

102 graphical user interface

103 memory

104 processor

110 graphical item

120 graphical pointer

121 graphical pointer in locked state

122 initiation position of the graphical pointer

130 eye focus position

131 eye focus position with an active gravity effect

210 user eye gaze

220 initiation user gesture

230 displacement user gesture

231 displacement direction

260 engagement user gesture

300 providing a graphical user interface

310 displaying a graphical pointer

320 detecting a user eye gaze

330 detecting a displacement user gesture

340 controlling a rate of movement of the graphical pointer

350 detecting an engagement user gesture

1. A method for controlling a combined eye and gesture tracking system(100), wherein the method comprises the following steps: providing agraphical user interface (102) to a user including a plurality ofgraphical items (110) displayed on the graphical user interface (102,300); displaying a graphical pointer (120) on the graphical userinterface (102, 310); detecting a user eye gaze (210) associated withthe eyes of the user (320); and detecting a displacement user gesture(230) associated with a body part of the user (330); and controlling arate of movement of the graphical pointer (120) on the graphical userinterface based on the displacement user gesture (230) and the user eyegaze (210, 340).
 2. The method according to claim 1, further comprising:detecting that the displacement user gesture (230) corresponds to one ofa plurality of predetermined displacement user gestures associated witha body part of the user, wherein the body part of the user is moved in adisplacement direction (231) relative to the graphical user interface(102); and controlling the rate of movement of the graphical pointer(120) comprises moving the graphical pointer (120) with a variable rateof movement in the displacement direction (231) to a position associatedwith one of the plurality of graphical items (110) taking into accountthe user eye gaze (210).
 3. The method for controlling a combined eyeand gesture tracking system (100) according to claim 1, wherein thedisplacement gesture is a motion of a body part in free space, which isdetected by a contactless optical device.
 4. The method for controllinga combined eye and gesture tracking system (100) according to claim 1,wherein detecting a user eye gaze (320) comprises: determining that theuser is looking at the graphical user interface (102); and determiningan eye focus position (130) on the graphical user interface (102). 5.The method for controlling a combined eye and gesture tracking system(100) according to claim 4, wherein the graphical pointer (120) is movedonly if the displacement direction (231) is directed in a directionwhere the eye focus position (130) is located.
 6. The method forcontrolling a combined eye and gesture tracking system (100) accordingto claim 4, wherein controlling a rate of movement of the graphicalpointer (120) comprises: comparing the displacement direction (231) withthe eye focus position (130); and moving the graphical pointer (120)with a higher speed, only if the displacement direction (231) correlateswith the eye focus position (130).
 7. The method for controlling acombined eye and gesture tracking system (100) according to claim 1,wherein each graphical item (110) comprises a predetermined snap-in areaaround the graphical item, wherein a rate of movement of the body partleads to a smaller rate of movement of the graphical pointer (120) whenthe graphical pointer (120) is within the predetermined snap-in area andthe displacement direction (231) is not substantially in direction tothe graphical item (110), compared to a rate of movement of thegraphical pointer (120) when the graphical pointer is within thepredetermined snap-in area and the displacement direction (231) issubstantially in direction to the graphical item (110) or when thegraphical pointer (120) is not within the predetermined snap-in area. 8.The method for controlling a combined eye and gesture tracking system(100) according to claim 1, wherein the method further comprises:detecting an initiation user gesture (220); determining if theinitiation user gesture (220) is substantially equal to a predeterminedinitiation user gesture; wherein the steps of displaying a graphicalpointer and controlling a rate of movement of the graphical pointer areperformed only if the initiation user gesture (220) is substantiallyequal to the predetermined initiation user gesture.
 9. The method forcontrolling a combined eye and gesture tracking system (100) accordingto claim 1, wherein displaying the graphical pointer (310) furthercomprises: determining an initiation position (122) of the graphicalpointer (120) on the graphical user interface (102); and displaying thegraphical pointer (120) at the initiation position (122).
 10. The methodfor controlling a combined eye and gesture tracking system (100)according to claim 9, wherein the initiation position (122) of thegraphical pointer (120) is the eye focus position (130) on the graphicaluser interface (102); or the position on the graphical user interface(102), which the user has been looking at for the longest time within apredetermined time range; or the position on the graphical userinterface (102), which the user has been looking at with the highestcount of refocusing actions within a predetermined time range; or amedian of determined eye focus positions (130) that have been determinedwithin a predetermined time range, or a predetermined position on thegraphical user interface.
 11. The method for controlling a combined eyeand gesture tracking system (100) according to claim 1, wherein themethod further comprises: detecting an engagement user gesture (260)associated with the body part of the user (350); determining, if theengagement user gesture (260) is substantially equal to a predeterminedengagement user gesture; and selecting the one of the plurality ofgraphical items (110) associated with the position of the graphicalpointer (120), if the engagement user gesture (260) is substantiallyequal to a predetermined engagement user gesture.
 12. The method forcontrolling a combined eye and gesture tracking system (100) accordingto claim 1, wherein the method further comprises setting the graphicalpointer (120) to a locked state (121) during a transformation betweenuser gestures of the body part or during the execution of a user gesturethat is not authorized to change the position of the graphical pointer(120) in such a way, that during the locked state (121), the position ofthe graphical pointer (120) cannot be substantially changed from thelast valid position of the graphical pointer (120) until a user gestureis recognized, which is authorized to change the position of thegraphical pointer (120).
 13. The method for controlling a combined eyeand gesture tracking system (100) according to claim 1, wherein the bodypart comprises at least one of a finger, in particular index finger,part of a hand, a hand, part of an arm, an arm, or part of a shoulder ofthe user.
 14. The method for controlling a combined eye and gesturetracking system (100) according to claim 1, wherein the method furthercomprises: detecting, that a user gesture made by the user is notsubstantially equal to one of the group of predetermined initiationgesture, predetermined displacement gesture or predetermined engagementgesture; and removing the graphical pointer (120) from the graphicaluser interface (102).
 15. A combined eye and gesture tracking systemcomprising: at least one camera (101), at least one graphical userinterface (102) a memory (103) configured to store program code, atleast one processor (104) coupled with the memory (103) and the at leastone graphical user interface (102) and configured to execute the programcode, wherein execution of the program code causes the at least oneprocessor (104) to perform the following: displaying a plurality ofgraphical items (110) on the graphical user interface (102) to a user(300); displaying a graphical pointer (120) on the graphical userinterface (102, 310); detecting a user eye gaze (210) associated withthe eyes of the user (320); and detecting a displacement user gesture(230) associated with a body part of the user (330); and controlling arate of movement of the graphical pointer (120) on the graphical userinterface based on the displacement user gesture (230) and the user eyegaze (210, 340).